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GENERAL INTRODUCTION TO THE P.E.P. REPORT 1969-1973 



The P.E. H Report 1969-1973 focuses on the various findings and activities of the Pv .gram Evaluation 
Project. It is being published in pamphlet form, with one pamphlet for each chapter. 

As of January, 1974, the Program Evaluation Project is funded by a three year collaborative grant 
with the Mental Health Services Division of the National Institute of Mental Health. The purpose of the 
grant is to emphasize the coordination and dissemination of information on a variety of program evaluation 
methodologies. Currently, it is expected that the title of the organization will be changed to the Program 
Evaluation Resource Center during 1974. 

Further information on the Goal Attainment Scaling methodology and program evaluation is available in 
other written and recorded materials from the Program Evaluation Project office. Currently, Chapter One 
on the "Basic Goal Attainment Scaling Procedures" and Chapter Three on "An Introduction to Reliability and 
the Goal Attainment Scaling Methodology" are available. 
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AJ..H . A J. x--"^--- •^K-^"*' VI uii iin.cyiatcu, }jr gyrdiu evaiUdtlOn SVSiem TO 

he^^lt Outpatient Program, Hennepin County Mental Health Service, Minneapolis, Minnesota. This evalu 
tlon system is an extension of and expansion of the research and evaluation activities conducted bv the 
rogram Evaluation Project, 

rogress to date is reported on evaluation of the program and evaluation of the individual client Pro- 
ess and outcome feedback, data collection procedures and aids are also outlined. 



IJOR FINDINGS : The tasks accomplished in stage one of the Adult Outpatient Program Evaluation System de- 
ilopment included determining the scope and specifications for the evaluation system and piloting- the de- 
gn in one of the Program's three program components. h y 



a 

In- 



e evaluation strategy that was conceptual i::ed encompasses all of the Program's activities. Using 
stems analysis approach, the program components are treated as an integrated set of sub-systems 
rmation specifications require the use of both process and outcome variables. 

LnrtpS ^Iln^Knlh'^'''''^" component was selected for testing of the methodology. A goal statement was 
rmulated, and both process and outcome objectives were derived from it. Evaluation criteria were stated 
measurable terms for all objectives using the Goal Attainment Scaling method. Information require- 
nts were determined, instruments assembled and/or developed, and pilot data collected 




addition to the administrative application, the Adult Outpatient Program has retained Goal Attainment 
iling for evaluation of the individual client. Procedures are employed for the roitine review o^^^^ 
inical goal attainment by both the clinician and the program manager. 
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Q III: Goal Attainment Scores for Individual 
ii/^" and Group Psychotherapy Modalities 



Outpatient Program during the first year of the 



This chapter presents thi evaluation theory, 
methodology, and strategies that have been devel- 
oped and applied by the staff members of the Adult 
Outpatient Program of the Hennepin County Mental 
Health Service since termination of the data col- 
lection phase of the Program Evaluation Project 
In October, 1972. These new procedures are re- 
ferred to, for the sake of brevity, as the "Re- 
design". 

The Re-design, which is staffed and finan- 
cially underwritten by the Adult Outpatient Pro- 
gram, is a multi-dime isional evaluation system 
characterized by goal-oriented process and out- 
come measures; it encompasses evaluation of both 
the program itself and client progress in thera- 
py; and it has been integrated into ongoing ac- 
tivities of the program. 

At the time of this report, the Re-design 
is being applied only to the direct clinical 
services component of the program. Additionally, 
evaluation procedures are to be integrated into 
the other two components of the program, specif- 



ically, training activities and research/evalu- 
ation activities. Ultimately, the entire evalu- 
ation approach will be assessed to determine its 
practicality and usefulness. 



I- Overview: Hennepin County Mental Health 
Service . 

The Hennepin County Mental Health Service 
is a community mental health center located with- 
in Hennepin County General Hospital in downtown 
Minneapolis. Organizationally, the Mental Health 
Service is comprised of eight programs each of 
which delivers a variety of services. Service 
categories include: direct clinical services, 
training activities, community consultation and 
education services, and evaluation and research 
activities. Figure I summarizes the Mental Health 
Service organizational structure and the alloca- 
tion of professional staff to each program. 



FIGURE I: Organizational Structure of the Hennepin County Mental Health Service. 
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A. Adult Outpatient Program: Staffing^ Pattern , 



Mission Statement, and Program Components 



for the first year of the Re-design. 



Within the Mental Health Service, the Adult 
Outpatient Program is the largest outpatient pro- 
gram. 

1. Staffing Pattern. 

The staff is mul tidiscipl inary, including 
psychiatrists, psychologists, psychiatric social 
workers, and psychiatric nurses. There is the 
equivalent of 13 full-time positions: approxi- 
mately 10.5 positions are allocated to the di- 
rect clinical services component, 2 positions are 
appor'iioned to the training component, and .5 of 
a pos'tion is dedicated to the evaluation and 
research component. 

2. Mission Statement. 

It is the general mission of the Adult Out- 
patient Program to achieve the following: 

First, to arrest the progression of mental 
illness of residents who are identified as suf- 
fv.ring from mental illness, to reintegrate resi- 
dents into their psycho-social milieu, and to 
establish productive, independent living patterns, 
by offering mental health services to Hennepin 
County residents; 

Second, to augment mental health manpower, 
by offering practicum experiences to students 
preparing for professional and paraprofessional 
mental health roles; 

Third, to program optimally for the delivery 
of mental health services and practicum experi- 
ences, by using the results of disciplined in- 
qui ry . 

3. Program Components. 

The Adult Outpatient Program considers each 
major activity domain, such as direct clinical 
services, a program component. A brief func- 
tional analysis of each program component follows: 

Direct Clinical Services Component . The 
Adult Outpatient Program makes services available 
to all residents of Hennepin County, regardless 
of presenting problem, degree of dysfunction, 
history of prior psychiatric treatment, socio- 
economic status, ability to pay, geographic lo- 
cation, or lastly, the availability of other 
service-providers. If a client states a strong 
preference for receiving services from the pro- 
gram. As a result, the p^^ogram staff may respond 
to all major diagnostic groups and all life di- 
lemmas, and they may render both diagnostic and 
treatment services. Diagnostic services include: 
telephone screening and referral, Individual and 
group intake, psychological testing, and medical- 
psychiatric consultation. The treatment modali- 
ties Include: indl-^^idual and group psychotherapy, 
marital and family counseling, behavior therapy, 
and chemotherapy. The scope of the direct clin- 
ical services component is detailed by Table I, 
which reports the actual service transactions 



TABLE I: Direct Clinical Services' Transactions 
Provided During the First Year of the 
Re-Design. 



TYPE OF TRANSACTION 


MONTH! V 
AVERAGE 


FIRST VFAR 
TOTAL 


Telephone Screening 
and Referral 


211 


2,535 


Intake Interviews 


143 


1 ,719 


Psychological Testing 


93 


1,120 


Treatment Interviews 


864 


10,369 


Total s 


1 ,311 


15,743 



Training Component* Clinical training is 
provided for students of the traditional mental 
health disciplines and for practitioners In new- 
ly emerging fields, such as paraprofesslonals . 
To this end, the Adult Outpatient Program has 
liaisons with the University of Minnesota and 
several other colleges and junior colleges. Un- 
dergraduate students in social work, sociology, 
and psychology, psychiatric residents, and grad- 
uate students In psychology, social work, public 
health nursing, family lifej and vocational coun- 
seling are integrated into and rotate through the 
Adult Outpatient Program. Approximately 50 stu- 
dents participate in the program each year. 

Evaluation and Research Component. All pro- 
fess 1onaT~TtafriTiTTIT"stu3eni^^ in 
the program evaluation activities which will be 
outlined in the remainder of this chapter. In 
addition, specialized evaluations have been con- 
ducted and are being conducted by some staff. 
Although no time is formally allocated to research, 
some staff have researched and are researching 
concerns of special interest to them. 
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II . Antecedents of the Re-Design 



III. Evaluation of the Program 



The original Program Evaluation Project re- 
search was sponsored by a grant from the National 
Institute of Mental Health. Its purpose was to 
assess by means of Goal Attainment Scaling! the 
relative efficacy of four psychotherapy modes In 
a community mental health center setting. During 
the course of this study, much of the data col- 
lected for research purposes was also made avail- 
able to the Mental Health Service, and especially 
to the Adult Outpatient Program, for program 
evaluation needs. 

In June, 1972, the Program Evalua^^on Project 
announced it would terminate its data collection 
procedures In the Mental Health Service during 
October, 1972. At this time, the staff of the 
Adult Outpatient Program began to assume respon- 
sibility for its own program evaluation system. 
The reality of steadily Increasing demands from 
funding sources for program justification and 
accountability was a further stimulus for de- 
veloping and upgrading the evaluation system 
innovated by the Program Evaluation Project. The 
first phase of the Re-design was implemented in 
November, 1972. 

The Re-design draws very heavily upon pre- 
cedents established by the Program Evaluation 
Project, including use of Goal Attainment Scaling 
for administrative goal setting and as a clinical 
outcome measure. The purpose of the Re-design 
is twofold: to evaluate the effectiveness and 
efficiency of the Adult Outpatient Program rel- 
ative to its stated goals; to demonstrate that 
program performance can be maintained or improved 
on effectiveness and efficiency measures through 
the use of evaluative data. Primary audiences 
for the information produced from the Re-design 
are; (1) the professional staff of the Program, 
(2) the administrators of the Mental Health 
Service, and (3) the funding sources. Clients 
from the population at risk, as well as the 
aforementioned groups, should benefit from im- 
proved program performance. 



Goal Attainment Scaling provides a aoal- 
setting format which essentially consists" of set- 
ting a criterion level for treatment outcome, ad- 
ministrative outcome, etc. (i.e., depending on 
the application) and attainment levels above and 
below the "expected" criterion. Five attainment 
levels are scaled for each treatment goal. Where 
more than one goal exists they may be relatively 
weighted. The unigueness of the Goal Attainment 
Scaling technique is the setting of attainment 
levels and the calculation of an outcome score 
which has been developed with the technique. 
The outcome score has the statistical properties 
of a mean of 50 and a standard deviation of 10. 



A. Model Construction . 

In conceptualizing an evaluation model, pro- 
gram evaluation In terms of process variables 
alone was recognized as Inadequate. Such a process- 
variable evaluation model can generate only state- 
ments of this type: "18,000 services were rendered 
to 2,000 clients during 1973," This process model 
can address the question of how many services were 
provided but not the impact of services upon cli- 
ents and Is, therefore, severely limited as an 
evaluation approach. 

Program evaluation based exclusively upon 
outcome variables is also inadequate. Such an 
outcome-variable evaluation model can produce only 
"evaluative" declarations of this type: "65 per- 
cent of the treatment goals negotiated by the 
staff with clients were achieved," This evaluation 
model can address the question of the impact of 
services upon clients but not how many services 
were provided. 

Both process and outcome information are 
necessary for program evaluation. It should also 
be concluded that considering process and outcome 
variables as unrelated events ignores the rela- 
tionship between them. It is reasonable to 
assume, however, that there is an optimal volume 
of clients with which a professional staff can 
achieve optimal treatment success. To the extent 
such a relationship does exist, knowledge of it 
is critical for efficient and effective program- 
ming. The evaluation model discussed in this 
chapter is based on the premise that there is an 
interactive relationship between process and out- 
come variables. The theoretical model is outlined 
below. 

1. Theoretical Model, 

Knowledge of the interaction of process and 
outcome variables is postulated to provide the 
best possible information for evaluative decision 
making. Similarly, it is assumed there are crit- 
ical interactions among program components. These 
interactions should be especially significant in 
the Adult Outpatient Program because all staff 
members participate in all program components. 

The theoretical evaluation model is based 
upon the assumption that the interaction between 
process and outcome and the interaction among pro- 
gram components can be observed. For example, if 
we attempt to increase use of clinical time for 
screening interviews by abbreviating the Officer 
of the Day (telephone) contacts, but then note a 
decline in the rate of successful referrals ac- 
complished during these contacts, we will construe 
this as a possible interaction. That is, objec- 
tives can be established, the attainment level of 
each can be monitored, and the degree to which 
-iccomplishing one objective affects the others 
can be recognized. The postulated interaction of 
process and outcome variables is represented by 
the Venn Diagram in Figure 11(a), The relation- 
ship avHong program components is depicted by the 
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FIGURE II: Venn Diagrams illustrating the relationship between Process and Outcome variables and the 
relationship among the Adult Outpatient Program Re-design components which are assumed in 
the theoretical model. 



Figure Ila Figure lib 




Interaction of Training 

and Research & 
Evaluation Components 



Venn Diagram in Figure 11(b). The theoretical 
evaluation model is based upon the integration 
of the interactions shown in Figures 11(a) and 
!I(b) above. 

Performance of each program component is 
a function of process, outcome, and the inter- 
action between process and outcome. Overall 
program performance is the result of the per- 
formance in each program component and the in- 
teractions among components. 

2. Applied Model. 

Development of the applied model began 
with a critical review of the Adult Outpatient 
Program's purposes. A mission statement and 
goal statements for the program components 
were articulated. Objectives and sub-objectives 
were then deduced from the goal statements and 
specified in measurable terms. The relation- 
ship shown in Figure III illustrates the deduc- 
tive framework within which the sub-objectives 
were derived from ' le mission statement. 

The mission statement was derived from 
the enabling legislation for community mental 
health centers. Goal statements for the pro- 
gram components reflect the director's inter- 
pretation of the activity domains which the 
Program must participate in and be accountable 
for. These goal statements also pinpoint tiie 
critical evaluative concerns. Decisions re- 
garding which concerns were 'critical* were 
made by the professional staff and doubtlessly 
incorporate their aggregate values and priori- 
ties. Specific guidelines which might have 



dictated the critical evaluative concerns were 
not available. 



FIGURE III: Hierarchical relationship among 

mission, goal, objective, and sub- 
objective statements which is the 
logical framework of the applied 
model . 
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The Goal Attainment Scaling format is em- 
ployed to accommodate the objectives and sub- 
objectives, stated in measurable terms. An ex- 
pected level is established for each objective 
or sub-objective. Figure IV shows the Goal At- 
tainment Scaling format. As may be observed in 
Figure IV, scale values range from +2 for the 
"most favorable outcome thought likely" to -2 
for the "most unfavor;;ble outcome thought likely." 



FIGURE IV; Goal Attainment Scaling Format. 
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B. Progress to Date . 

At this time, the first phase of implementa- 
tion is nearly complete. The remainder of this 
section will focus on evaluation cf the direct 
clinical services component. 

The goal statement for the direct clinical 
services component is: to insure that an optimum 
volume of identified clients are efficiently pro- 
vided with appropriate and acceptable referral , 
diagnostic , and treatment services which assist 
them in acTiieying client-relevant , quality goals 
and with which they are reasonably satisfieQ . 
Those referents underlined in the goal statement 
were used to derive effectiveness and efficiency 
objectives. 

The effectiveness objectives include: 

I» To insure Ihdt referral services offered 
to persons who seek services are: 

A. Appropriate 

B. Acceptable 

IL To insure that diagnostic services pro- 
vided to cl ients are: 

A. Appropriate 

B. Acceptable 

III. To insure that treatment services pro- 
vided to clients are: 



Using the computational formula developed 
by Kiresuk and Sherman (1968), 

Tj = 50 + ^O^^i^i 

where it is recommended that p be taken as about 
.30. 

5 Goal Attainment score can be computed for 
each objective. Goal Attainment scores for 
each of several objectives can then be aggre- 
gated utilizing a modified formula derived by 
Sherman (1973). 

y. = 50 + £Wi(Ti-50) 

/(l-p)ZWj^ + p{ZWj)^ 

Where it may be necessary to select a value for p 
commensurate with the intercorrelations of the 
sub-objective scores, T j . 

Through this process of Goal Attainment score 
aggregation, as outlined above, a Program Per- 
formance Index Score can be generated. As 
with the Goal Attainment Score, the statisti- 
cal characteristics of the Index Score are a 
predicted mean of 50 and a standard deviation 
of 10 where a mean of 50 indexes programmatic 
performance (goal achievement) at the "expected 
level." Administratively, the Index Score pro- 
vides a useful frame of reference for analyzing 
overall programmatic performance. 



A. Appropriate 

B. Acceptable 

IV. To insure that treatment goals set for 
individual clients are: 

A. Personally relevant 

B. Clinically relevant 

C. Clinically realistic 

V. To insure that treatment goals are being 
achieved. 

VI. To insure that services rendered to cli- 
ents are satisfactory in terms of: 

A. Waiting time for services 

B. Scheduling 

C. Fees 

D. Outcome 

E. Services »^ece)ved 

F. Overall evaluation 

Efficiency objectives include; 

I. To insure that clients receive services 
in the shortest possible time. 

A. Screening interval 

B. Diagnostic interval 

C. Treatment interval 
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If. To insure that an optimum volume of 
identified clients are servei 

A. Screening services 

B. Diagnostic services 

C. Treatment services 

Examples of an effectiveness and an effi- 
ciency sub-objective are shown in Figures V(a) 
and V(b). 

The evaluation concern wnich is the vocus 
of Figure V(d) is: to insure that tne treat- 



ment goals of clients receiving individual 
psychotherapy are being attained. An asterisk 
indicates the level of achievement of all in- 
dividual psychotherapy clients in a particular 
month. The administrative criterion is that 
between 65 and 8C percent of these clients at- 
tain a clinical Goal Attainment score of at 
lea'it 50. (A clinical Goal Attainment score 
of 50 represents the accomplishment of treat- 
ment goals at the "expected level.") During 
five of the eight months sampled, this adminis- 
trative criterion was satisfied; during three 
months it was not. 



FIGURE V: Example Effectiveness and Efficiency Sub-objectives. 

V(a). Effectiveness Sub-objective Dealing with Achievement of ' reatment Goals. 
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Minimizing the time interval between a cli 
ent's request for service and his first sched- 
uled appointment is the evaluation concern rep- 
resented in Figure V{b). The administrative 
criterion is that the time interval, measured 
using median number of days, falls between four 
and seven days. Using an asterisk to indicate 
the outcome for a particular month, the admini- 
strative criterion was -net during four months 
and surpassed during three months across the 



data collection period. 

For the direct clinical services compo- 
nent, the Program Performance Index Score for 
the first year of the Re-dssign was 54.0 This 
has been aggregated ^rorr. the outcome scores 
of eight objectives and 21 sub-objectives of 
thp direct clinical services component. Figure 
VI on the following page overviews the process 
of generating the Index Score. 



FIGURE V: Example Effectiveness and Efficiency Sub-objectives. 

V(b). Efficiency Sub-objective Oealing with Client Waiting Time Between First Contact (Service Request) and 
Scheduled Intake Appointment. 
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and scheduled intake is 
between 8 and 12 days. 
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* 






This Month's Index Score 50 

Data Data 


60 70 60 


50 50 50 



13 
9 



c 

C7) 



oc 



T- to 

ZD D) 
O 
iO s- 
in o. 

O C 



go 

c »— 
OJ 3 
CD "O 

Of 
U -fJ 

CO 

«^ 

X o 
■o J- 

C flS 

>- 

OJ 

c to 
o 

(4. OJ 
s- ^ 

OJ 4J 

E 
E O 

10 s. 
J- ^- 

D) 

o 10 



O OJ 



OJ 4J 

il 

> 3 




OC 
ZD 
O 



d) (i) 

(g) (g) 





H 


0 




> 


VO 








LO 




> 










cn 




> 
H 


ro 






M 
M 


a> 




vo 






M 
M 


<yi 




uo 






M 


0 







(s) (s) 



S J V S 



(5) (2) <c) 




X 




T3 


to 




O) 
X3 




OJ 1 
f- ^ » 


£! -0 OJ 




c 




0 ^ 


0 OJ > 








Q.-M to 


to H- -•-» 








flO to 


•f- 0 










-M "O OJ 




■M 




(/) 2 


C 0 '1-5 








•r~ 0 (/) 












C 




10 r— 0 


C OJ 








OJ U 


f- ^ 




-M 




13 0 f- to 


ro 4J . 




(O 




E 

U OJ -M 










-fJ C7^ cn 




3 




0 > ^ C 


(O C C 




U 
















OJ to "O 




10 




CO u ^ c 


> 3 r— 




0 




3 -Si* "5 


•1- OJ 
4J -O ••- 




c: 




^ r- 4J 


U OJ 




*r- 




r- 0 OJ -M 


OJ 4J 








10 i > n 


•1-5 10 to 


OJ 


■0 




C ^ OJ 


^ air— 




OJ 




f- 3 r— OJ 


0 0) 3 

1 C E 


0 


to 




cr> to > 




3 




•r- OJ T- 
OJ > -M 


3 C7) 0 


to 


to 




0 ^ f- 0 




X 


Q. 


■M 4J OJ 




OJ 


OJ 


0 


OJ 0 


OJ CO "O 


■M 


0 


^ 0 OJ ^ 


i5 roS 


c 


00 


to 


}_ 4J 0 





X 
OJ 

■o 



> OJ 



4J OJ 'r- 


c 


to fO CZ'f- 






OJ 


CTii — 10 1 — 






0 


C7) C 0 -M 




^ CJ>T3 C 






0 Cr— OJ 




cn 0 0 




f- OJ T- 




ro W OJ 




OJ to *r- 0 


Ul 


OJ 




-C 3 >,.r- 




C CO Q_ 




4J 


•0 


1o3 E ^ 






c: 




-M CD 1 — LU 


(O 


C7) 10 OJ 




ro -M 3 




«0 T3 J- ^ 


-M 


« E -O 
to CD C 


to 


OJ O) 4-> 


C 


to 


OJ 0 


OJ 


g, g, 0 « 


OJ 




c: 


c: 




0 


0 cr» to 


OJ 


T3 «+- 


D- 


0 o>co to 


> 


(/) 0 OJ 


E 


(/) (O ^ ^ 






0 
0 


X 0) OJ 
QJ C -O > 




0 OJ 0 




OJ 


0 -c 0 


to 


■0 <0 OJ 1- • 




to +-> c: 00 


OJ 


C -M to 








1— ^- 0 qj 

OJ«r- OJ C. 


UJ 


X o» *5 X 




OJ C r— OJ 


> 


0) > -0 0 


QJ ^3 -r- QJ -O 




^ OJ 0 0 




C to -r- C 


OJ 


J— • — 2E UJ t/) 


1— 


•f- 3 


CO 



5 £ > 

I I 



O I 

= s . 

4J r- OJ OJ 
fO » — TD to 
^ O OJ 

4J o a x: 

to fO 

OJ OJ a 

■M fO ^ C 

fO "O -»- 

U OJ 

•f- J- o 

-O OJ «0 4J 

C r- 

T- ^ </) CZ 

(O OJ Q 

OJ E 

J- r- 3 

3 «TJ •p 

C71 > OJ 

•t- «TJ O 

*»-. O 

c o. 



J- 

o +J 



OJ 



< -M o 
Z -O 4J 



Q. O 
O I 
»— ^ 
OJ 3 
> «/) 



ERIC 



10 

14 



IV. Evaluation of the Individual Client. 

A. Model Construction . 

The model employed for evaluation of the 
clinical case is based upon a modification of 
the "Four Mode" Goal Attainment Scaling pro- 
cedure developed during the Program Evaluation 
Project. Greater emphasis is being placed on 
client participation in the goal -set ting pro- 
cedure. 

1. Theoretical Model. 

Goal Attainment Scaling allows idiographic 
goal setting for the individual client. The 
first assumption made is that a value can be 
associated with a client's position on a goal 
dimension. It is also assumed that in using 
such values an index score can be derived which 
reflects a client's overall status on any num- 
ber of goal dimensions. 

2. Appl ied Model . 

AH clients assigned to ongoing, individual 
and group psychotherapy are being evaluated. 
Since these two modalities account for more than 
50 percent of all treatment provided, evaluation 
of the clients receiving these services was un- 
dertaken first. Goal Attainment Follow-up Guides 
are constructed by an intake worker at the close 
of the intake Interview(s). Consumer partici- 
pation in goal setting is being emphasized in 
the applied model. All goals are set with a 
three month follow-up date. 

The therapist assesses the clinical rele- 
vancy and realism of goal setting done during 
the intake interview(s) . This critique of follow- 
up guide content is accomplished within two weeks 
after the client is assigned to the therapist for 
ongoing therapy. Results of the critique pro- 
vide data for evaluation of the program, specif- 
ically in terms of quality control of goal set- 
ting. Follow-up is conducted and outcome evalu- 
ated by the therapist after three months of 
therapy. Utilization of the therapist as the 
follow-up evaluator is being explored in the 
program, given the constraints of a very limited 
budget for independent (or contractual) follow- 
up. But because the feasibility and reliability 
of this alternative to independent follow-up has 
not been previously established at the Mental 
Health Service, outcome scores generated by 
therapists are being verified through the use 
of corroborative interviews. 

B. Procuress to Date . 

In discussing progress to date, reference 
should be made to Figure VII which offers a 
schematic overview of the direct clinical ser- 
vices delivery system. 

1. Entry to the Delivery System. 

Clients enter the delivery system via either 
the Crisis Intervention Center or the Adult Out- 



FIGURE VII: 



Schematic Overview of the Adult 
Outpatient Program Direct Clinical 
Services Delivery System. 
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patient Program telephone screening and referral 
service (Office of the Day phone), (a report on 
the activities of the Crisis Intervention Center 
may be fo d in the Crisis Intervention Center 
chapter of the P.E.P. Report 1969-1973. 1 Once 
clients who may require ongoing treatment services 
are identified from either of these two sources, 
intake interviews are scheduled. Approximately 
33 percent of the clients scheduled for intake in- 
terviews are referred from the Crisis Intervention 
Center, with the remaining 67 percent identified 
by the Office of the Day telephone service. 

2. Diagnostic or Intake Interview Process. 

Depending upon the client*s preference, in- 
take interviews may take place individually or in 
a group. All professional disciplines perform in- 
take functions. Table II gives the relative per- 
centages of intake activities by professional dis- 
cipline for the first year of the Re-design. In 
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TABLE II: Distribution of Intake Activities by Professional Discipline and Type of Intake During the First 
Year of the Re-Design. 



INTAKE CLINICIAN 
DISCIPLINE 


INTAKE 


TYPE 


TOTAL PERCENT 


INDIVIDUAL 


GROUP 


OF ALL 
INTAKES 


Psychiatry* 
Staff 
Students 


Percent 

2% 

6% 


Percent 


2% 
6% 


Psychology 
Staff 
Students 


^s% 

6% 


18% 

6% 


Social Work 

OLaTT 

Students 


23% 
S% 


25% 
1% 


6% 


Psych. Nurse 
Staff 


14% 


14% 


TOTAL 


74% 


26% 


100% 


*Psychiatry staff and students are available 
during the Intake process. 


for medical psychiatric consultation 



intake, 1,168 new clients were seen and on the 
average received 1.4 intake interviews. 

Both medical-psychiatric consultation and 
psychological testing are available to the intake 
worker during the intake process. Use of these 
services is siiirmarized as follows: 85 percent 
of the new clients completed routine, psycho- 
logical testing; 33 percent of the new clients 
were provided with a medical-psychi atric con- 
sultation. 

The intake process culminates with an in- 
take summary. This suntnary includes the client's 
presenting complaints, the stresses he is con- 
fronted with, his independent efforts to accom- 
modate, any history of previous treatment* his 
personal (education, vocational* social, etc.) 
history, the results of medical-psychiatric con- 
sultation and psychological testing, the diag- 
nostic impressions, and a consensual treatment 
plan. 

Intake sunmaries and follow-up guides are 
forwarded to the Intake Committee, comprised of 
senior staff, who review all cases recommended 
for treatment. In addition to reviewing clini- 
cal judgment, this coimittee implements treat- 
ment assignments according to the following pro- 
cedures. A record of therapy openings, listing 
which therapists on the Adult Outpatient Program 
staff are available for specific treatment modes. 



is updated each week by . ^ earch clerk. This 
information is used by c - ;mmittee to assign new 
clients to therapy modes aiiu therapists. 

A randomized assignment procedure is avail- 
able to the committee for a client If individual or 
group therapy Is recommended. This randomization 
procedure consists of two operations. The first, 
employed If it Is the Intake worker *s clinical 
judgment that the client may benefit equally from 
either individual or group treatment, involves ran- 
dom assignment between these modes; this Is accom- 
plished by drawing the first card from a randomly 
shuffled deck of available therapy mode cards. 

The second operation is used to select a thera- 
pist. A deck of randomly shuffled "available thera- 
pist" cards Is prepared for Individual therapy and 
a second is prepared for group therapy. The first 
card of the appropriate deck is drawn, and 1t spec- 
ifies the therapist who will receive the case 
assignment. All of these assignment procedures can 
be aborted if an obviously unworkable assignment, 
in the judgment of the committee, results. 

Experience to date has revealed that approxi- 
mately 85-90 percent of all intake worker treatment 
recommendations are implemented without change. 
Table III summarizes the dispositions implemented 
by the Intake Committee and reflects the intake 
workers' utilization of the various randomization 
al ternatives. 
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TABLE III: Intake Committee Case Assignment Summary for First Year of Re-Design. 



THERAPY MODF 


RANDOM MODE ASSIGNMENT PROCEDURE 


NON-RANDOM W 


ASSToNMENT PROCEDURE 


TOTAL 


RANDOM THERAPIST 


NON-RANDOM THERAPIST 


RANDOM THtKAr .'j 


'''ON-RANDOM THERAPIST 


ASSIGNMENTS 


Individual* 


24 


7 


81 


] 7 




Group* 


16 


12 


21 


73 


122 


Crisis Group 








67 


67 


Medication 












Therapies** 






— • 


116 


116 


Mdrrlage/Fcnlly 








81 


81 


Counseling 












Assignment to Another 






— 


79" 


79 


MHS Treatment Program 










Termination After 








204 


204 


Intake Only 










Assignir-ent Diipositior 








219 


210 


r.'ct Yet Comple'^ed 








TOTAL NEW CASES 


40 


19 


102 


797 


1168 



*Random assignment options are only applicable to individual and group psychotherapy modes 
♦Includes brief psychotherapy contact with a psychiatrist 



As indicated in Table III, shown above, 478 
cases were assigned to the individual and group 
psychotherapy programs during the first year of 
the Re-design (includes crisis group). Ideally, 
a follow-up guide should have been received on 
each of these cases. About 77 percent of the 
follow-up guides for guide-eligible cases were 
retrieved. The loss rate of 23 percent was 
attributable to a combination of administrative 
and implementation factors. 

Analysis of individual client data begins 
with content classification of the goals on the 
client's follow-up guide. The average number of 
scales per follow-up guide is three. Table IV 
provides the distribution of scales classified 
by content types for the Re-design evaluation 
sample, using problem classification guidelines 
prepared by Garwick ar:d Lampman (1972). Inter- 
personal and work problems are the most fre- 
quently occurring in this sample. However, 
even these account for only 22 percent of all 
problems identified by intake workers and their 
clients. This suggests that the problems which 
bring clients to a mental health center are 
quite varied in terms of content areas. 



TABLE IV: Classification of Treatment Goals Set 
During the First Year of the Re-Design 



Classification 


Percent of All Scales 


Aggression 


3% 


Alcohol Use 


3% 


Anxiety 


6% 


Decisions 


A% 


Depression 


9% 


Drug Use 


0%* 


Education 


2% 


Rami ly/Marital 


7% 


Financial 


0%* 


Interpersonal 


^^% 


Relationships 




Legal 


2% 


Living Arrangement 


3% 


Physical Complaints 


8% 


Psychopathological 


A% 


Symptoms 




Self References 


6% 


Sexuality 


6% 


Suicide 


4X 


Treatment 


7% 


Work 


m 


Miscellaneous 


4 


Total 


100^^ 
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3, Treatment Process. 

In an effort to control the quality of goal 
setting and to stimulate dialogue between intake 
workers and therapists, follow-up guide critiques 
are requested from therapists two weeks after 
case assignment. This follow-up guide critique 
process involves the therapist in appraising the 
relevancy and realism of goals and expected levels 
of attainment established'by the intake worker 
and the client during the intake interview(s). 
Data obtained indicate that 92 percent of the 
scales are judged relevant and 76 percent of these 
relevant scales are deemed to have realistic at- 
tainment levels, assuming a three month follow-up 
date. At the time of the critique, the therapist 
also has the option of adding new sca'^es in nego- 
tiation with his client. Such ^hv:'.rapist initi- 
ated goals are also being evaluated H the three 
month follow-up. The scores yielded are being 
analyzed separately from scores yielded by in- 
td^tt worker/client goals, and are not included 
i'-'- the analyses reported below. 

With the return of the follow-up guide cri- 
tique, no specific evaluation activities are con- 
ducted until the three month follow-up. At the 
designated follow-up date, the therapist receives 
a follow-up packet, in which he is asked to com- 
plete a therapy transaction summary, that is, a 
visit count, and outcome ratings, including the 
scoring of the follow-up guide. These follow-up 
materials are to be completed for all clients who 
have attended a minimum of two therapy sessions. 



Nine months of outcome data have b^en col- 
lected at the time of this report, and 253 clients 
have *-eached their follow-up date. Seventy-five 
percent of 190 clients have satisfied the follow- 
up criterion of attending at least two treatment 
sessions. Of this, follow-up sample, 173 clients, 
or 92 percent, were successfully followed-up, 

4. Characteristic: of the Outcome Score. 

This section deals with the outcome results 
obtained from the sample of 173 c'^ients success- 
fully followed-up. The process of an-ilyzing the 
outcome results begins with an examination of the 
score distribution. Figure VIII shows the Goal 
Attainment score distribution of the Re-design 
sample. This distribution is characterized by a 
mean of 52,5 and a standard deviation of 11,9, 
which closely approximates the statistics of the 
theoretical distribution (a mean of 50 and a stan- 
dard deviation of 10) described by Kiresuk and 
Sherman (1968). 

More critical analysis of the outcome scores 
required review of the Goal Attainment score's 
dependency upon other variables. Of first concern 
are guide construction variables, that is, mode 
of intake and intake worker. As regards mode of 
intake as a possible source of prediction bias in 
follow-up guide construction, the data shown in 
Table V on page 17 indicate no difference in the 
mean outcome score of follow-up guides developed 
in individual and group intake; this suggests that 
the mode of the intake interview has no signifi- 
cant effect on the Goal Attainment score. 



FIGURE VIII: Histogram of Goal Attainment Score Distribution for Therapist Conducted Follow-ups During 
the First Year of Re-Desiqn. 



X 

X X 
X X 
X X 
X X 
X X 



X X 
XX XX 



X 
X 
X 
X 
X 
X 
X 
X 

xxxx 
xxxx 
xxxx 
xxxx 
xxxxx 



X 
X 

X X 

XXX 

XXX X X 

XXX X X 

XXX XXX X 
X XXX X XX X X 
XXXXXX X XX X X X 
XXXXXX xxxx XXX XX 
XXXXXXXXXXX XXXXXX 



Number of Cases: 173 
Mean: 52.5 

Standard Deviation: 11.9 



XX 



X 
X 

X 
X 
X 

X X 
X X 



X 

w 



10 



20 



30 



40 



50 



60 



70 



90 



100 



Goal Attainment Scores 



14 

18 



TABLE V: Goal Attainment Outcome Statistics by 
x)de of Intake for the First Year of 
Re- Design. 



the sample sizes are small, the preliminary re- 
sults shoWft with one exception, all scores in 
the "expected level" goal attainment ranqe. 



Intake 


Number of 
Cases 


Mean Goal 
Attainment 
Score 


Standard 
Deviation 


Group 


55 


52.8 


11.5 


Individual 


118 


52.5 


12.2 


TOTAL 


173 


52.6 


11.9 



TABLE VII: Goal Attainment Scores for Adult Out- 
patient Therapists Who Have Completed 
More than Five Therapist Follow-ups. 



Theoretically, outcome scores on follow-up 
guides produced by each intake worker should dem- 
onstrate a mean of 50 and a standard deviation 
of 10. The statistical characteristics of the out- 
come score distribution of intake workers with 
five or more cases are entered on Table VI. Pre- 
liminary analysis reveals no statistical differences 
among the mean outcome scores of intake workers 
shown in Table VI. However, there is an indica- 
tion that several of the intake workers may be 
pessimistic in their goal setting. At present, 
a procedure which will allow for adjustment of in- 
take worker "bias" is being developed by the Pi - 
gram Evaluation Project staff which, when com- 
pleted, will be adapted for use in the Re-desig . 

TABLE VI: Goal Attainment Scores for Adult Out- 
patient Intake Workers with More than 
Five Intake Cases that Have Been 
f I lowed Up. 



Intake 


Number of 


Mean Goal 


Standard , 


Worker 


Cases 


Attainment 


Deviation 






Score 




A 


24 


52.16 


10.96 


B 


10 


52.10 


12.95 


C 


16 


57.50 


16.03 


D 


8 


53.75 


7.00 


E 


14 


54.69 


12.55 


F 


16 


57.06 


14.21 


G 


14 


55.78 


10.34 


H 


14 


49.42 


10.98 


I 


9 


51.11 


13.27 



Therapist 


Number of 


Mean Goal 


Standard 




Cases 


Attainment 
Score 


Deviation 


A 


11 


53.18 


8.98 


B 


7 


66.57 


10.96 


C 


8 


56.25 


8.15 


D 


7 


46.71 


9.79 


E 


27 


47.88 


9.64 


F 


10 


50.98 


10.58 


G 


10 


50.00 


8.43 


H 


21 


53.19 


14.61 


I 


16 


54.75 


9.65 



Score verification is accomplished through 
corroborative interviews conducted by mental health 
professionals not affiliated with the Adult Out- 
patient Program. These corroborative interviews 
are conducted via telephone on a random sample of 
cases followed up by therapists; this independent 
contact by the corroborative interviewer permits 
scoring the follow-up guide and completing a con- 
sumer satisfaction inventory. The mean of the 
corroborative interviewers' outcome score distri- 
bution is 52.5 and the standard deviation is 12.18, 
as compared to a mean of 54.2 and a standard devi- 
ation of 10.55 for the therapist distribution. The 
mean of the distribution of outct^me scores gener- 
ated by the corroborative interviewers is not sig- 
nificantly different than that of the therapist 
distribution. 

This demonstration of credibility for the 
therapist-generated outcome scores justifies their 
use in analysis of program variables. For example, 
the relative efficacy of treatment modes might be 
contrasted. The means and standard deviations of 
the outcome scores for individual and group psycho- 
therapy modalities are shown in Table VIII. Be- 
cause the sample sizes are modest, no attempt has 
been made to analyze separately for randomly-assiqned 
versus nonrandomly-assigned cases. Aggregate analy- 
sis demonstrates no significant difference between 
individual and group psychotherapy at this moment. 
(Table VIII appears on the following page.) 



Because the Re-design utilizes therapists as 
follow-up evaluators of their own treatment cases, 
the distribution of their outcome scores is of 
particular interest — the specific concern being 
that therapists in a self-evaluation system might 
inflate outcome scores. However, the statistical 
characteristics of therapist outcome score distri- 
butions are presented in Ta^^'e VII, and although 
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TABLE vIII: Goal Attainment Scores for Individual 
and Group Psychotherapy Modalities 
During the First Year of the Re-Design 



Treatment 


Number of 


Mean Goal 


Standard 


Mode 


Cases 


Attainment 


Deviation 






Score 




Individual 


101 


54.20 


12.38 


Group 


72 


50.31 


11.20 


TOTAL : 


173 


52.60 

1 


11 .90 

1 



V. Feedback . 

Fundamentally, the purpose of the Re-design 
is to assist in improving the performance of the 
overall program relative to its stated goals and 
the performance of professional staff members in 
the delivery of clinical services. To have the 
desired impact, feedback of evaluative information 
is imperative. 

Evaluative data on programmatic performance 
has been monitored by the director of the Adult 



Outpatient Program during the first year of the 
Re-design. Management decisions have been facili- 
tated through the use of this data. For example, 
as a reflection of the established fail rate ratio, 
over-scheduling is now being used to maximize pro- 
fessional staff time devoted to intake interviewing. 
Parallel investigation on "fail" rate problems is 
currently in progress. Much of this information 
has been shared with the professional staff through 
a series of informal progress reports and discus- 
sions at routine administrative meetings. During 
the next year, feedback of evaluative data on pro- 
grammatic performance will be compiled and routine- 
ly disseminated to the professional staff. 

Using therapists as follow-up evaluators on 
their own treatment cases insures clinical feedback. 
With completion of the follow-up packet, the thera- 
pist, iq prpqpnt.Prt with thp cniirjsf and outcome of 
treatment. 

More rigorous and systematic rejedback- proce- 
dures are being developed for information sharing 
with the professional staff on bcth the overall 
program and the individual clierc. I'ihen possible, 
data will be presented to the individual sviff mem- 
ber on his own performance.. Arjoregata information 
on the performance of prolessiorial coMeagues will 
also be provided and should facil ft-te ♦'evaluative 
data interpretation by rne in:'i vic'iMi :xUff member. 

The first ohsse of r.uc^i a sjstm has been im- 
plemented. A Sunple process feedback; raport deal- 
ing with intake CrJseloa'J is displayed in Figure IX. 



FIGURE IX: Sample Adult Outpatient Program Process Feedback Report 
Which is Generated for the Individual Intak'd Clinician. 
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The report summarizes the intake activities of 
November 8, 1973. Aggregate information indicates 
that a total of 13 cases were presented for ongoing 
treatment assignment with an average delay of 
9.2 days from initial interview until case assign- 
ment by the Intake Committee. Information ad- 
dressed to the specific intake worker includes a 
summary of the number of cases forwarded to the 
L'take Committee, average delay in doing so, and 
a client specific disposition summary. Also in- 
cluded is information on the status of the therapy 
openings within the program. 



VI. Data Collection and Processing . 

Data needed for evaluation of the program and 
for evaluation of the individual client are collect- 
ed routinely. Installation of a computerized re- 
porting facility, which will replace the previous 
"hand tally" methods and minimize the amount of 
clerical time involved in report generation, is 
under way. The basic data processing system which 
was developed during the Program Evaluation Project 
was adopted for the Re-design. However, several 
new reporting options have been implemented, such 
as feedback reports on process variables. 

The system was developed for use on an IBM 
360/370 computer for processing of data stored on 
magnetic tape. All processing routines are written 
in Fortran IV and are currently run in batch mode. 
Some features of the system include: client-orient- 
ed file structure; editing, updating, correcting 
and analyzing routines which are constructed for 
use by non-technical staff; flexible file structure 
content and file expansion capability; and security 
checks which protect against unauthorized access 
and/or the alteration or destruction of information 
by users of the system. 

In addition to the automated routines avail- 
able through the computer system, other computer 
outines have been built to assist in specific 
program evaluation and/or research data processing. 
These include the Minnesota Multiphasic Personality 
Inventory and the Shipley-Hartford processing rou- 
tines, cross-tabulation routines, and various 
statistical analysis routines. Using this modi- 
fied Program Evaluation Project processing system, 
a single data file is maintained which serves both 
evaluation and research purposes. 

The evaluation data base currently contains 
items which permit monitoring the attainment levels 
of both effectiveness and efficiency objectives es- 
tablished for the direct clinical services compon- 
ent. This data base allows monthly scoring of 
the attainment levels of both effectiveness and ef- 
ficiency objectives. The attainment levels are 
then used in the calculation of the Program Per- 
formance Index Score. The evaluation data base 
also contains items which pertain to the individual 
client, such as demographic information, trans- 
action or visit information, psychological testing 
results, follow-up guide content and follow-up re- 
sults, and termination information. These data 



can be used to evaluate the individual client and 
can be aggregated across clients to provide a 
characterization of the client population. 



SUMMARY. 



The evaluation system of the Adult Outpatient 
Program is called the Re-design to denote that it 
assimilates the four year's experience o» the clin- 
ical and research staff members who participated in 
the Program Evaluation Project. The Re-design, 
which utilizes Goal Attainment Scaling, undertakes 
evaluation of overall program and the clinical case 

With respect to overall evaluation of the Pro- 
gram, development of the applied evaluation model 
began with a critical review of the Program's pur- 
poses. A mission statement and program component 
goal statements were drafted. Lastly, objectives 
and sub-objectives were deduced from the goal 
statements and specified in measurable terms. 
Goal Attainment Scaling was used to scale achieve- 
ment levels for these objectives and sub-objectives 
Using this method, Goal Attainment scores can be 
generated which reflect the level of attainment of 
each objective and sub-objective. A Program Per- 
formance Index is derived through aggregation of 
these Goal Att;ainment scores. Progress to date 
has consisted cf model development and pilot in- 
stallation. Objectives and sub-objectives re- 
lated to the direct clinical services program 
component have been monitored and monthly Index 
Scores have been generated. Data derived from the 
pilot project will be utilized by the professional 
staff as base-rate information for assessing the 
utility of each objective and sub-objective as 
program monitoring measures. In addition, the 
base-rate data may be used in revision of the cri- 
terion levels of inappropriately scaled objectives 
and sub-objectives. 

Regarding evaluation o^ the individual patient, 
modifications have been effected in the procedures 
used during the data collection phase of the Pro- 
gram Evaluation Project. The design now hopefully 
fosters greater clinician and patient involvement. 
Treatment goal-setting through negotiation between 
intake workers and patients is being emphasized. 
Therapists are not only made aware of treatment 
goals, but they are asked to critique the goal- 
sett:«ng done at the time cf the intake interview 
and are encouraged to add treatment goals as dic- 
tated by further observation, again in consultation 
with the clients- Lastly, the design now utilizes 
the therapists as primary "follow-up worker" which 
insures they are presented with treatment outcome. 
Corroborative follow-ups, conducted by independent 
interviewers, provide data which is used to vali- 
date the outcome scores derived by therapists. 

For both overall program evaluation and eval- 
uation of the individual patient, data collection, 
data processing, and feedback systems are being de- 
veloped. Progress to date has included adaptation 
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of the computerized data storage and retrieval 
system developed by the Program Evaluation Pro- 
ject. In addition, several Re-design specific 
computer programs have been written to facilitate 
the processing of evaluation data. Various types 
of process and outcome feedback are currently being 
provided to the professional staff. Integration 
of several feedback reports has been undertaken. 

At this moment, conclusions about the merits 
and shortcomings of the Re-design cannot be def- 
initively stated. Experience to date promotes the 
professional staff's belief that the Re-design does 
represent an approach to evaluation which is com- 
prehensive and facilitates regular, systematic re- 
view of both the Program and the clinical case. 
The Adult Outpatient Evaluation System is an en- 
deavor designed to enlarge understanding of the 
Program, in terms of what makes it effective and 
efficient, and of the clinical case, in terms of 
what treatment works and how wel 1 . 

Goal Attainment Scaling, used in either eval- 
uation of a program or an individual client, is 
extremely flexible. In evaluation of a program, 
the manager selects those goals which are appro- 
priate for his program. He can set goals according 
to his value system and performance criterion levels 
which are desirable and realistic. Use of the Pro- 
gram Performance Index Score permits him to compare 
actual programmatic performance with desired per- 
formance. Similarly, in evaluation of an individual 
client, the clinician using any theoretical frame- 
work, can develop idiographic goals with his client. 
At follow-up, a Goal Attainment score can be com- 
puted which summarizes his client's status across 
all treatment goals. 

The aforementioned assets prompt optimism 
about the Re-design. However, potential liabilities 
exist in applications at both the program and clin- 
ical case level. At the program level, realistic 
scaling is necessary to minimize random fluctua- 
tions of the Program Performance Index score. Also, 
one must exercise caution in reacting to this and 
any index score. More experience with the index 
score concept is necessary before it can be utilized 
for major management decisions. Development of de- 
cision rules and guidelines is a must. At the in- 
dividual client level, reliability of Goal Attain- 
ment scores is a major concern. In the Re-design, 
in which therapists function as primary follow-up 
workers, the reliability of outcome scores may be 
affected by factors which are not operative in 
follow-ups conducted by independent Interviewers. 
As a result, provision for independent verification 
of therapist generated outcome scores and feedback 
of these corroborative scores to the therapists 
appears to be necessary. 

Given the merits and potential shortcomings, 
the critical demonstration of the Re-design's util- 
ity will be whether it provides evaluation data 
which result In either maintenance or improvement 
of performance. This utility will hopefully be 
dcmpnstrated at both the program and individual 
client level and will be assessed relative to the 
cost of the evaluation program. 

The next phase of Re-design activity will con- 
sist of: (1) reassessment of the direct clinical 



service goals and criterion levels for each ob- 
jective and sub-objective; (2) establishment of 
pilot-stage goals and scaling criterion levels for 
objectives and sub-objectives, on the training and 
rCfSearch/evaluat^on program components; and (3) 
completion of a cost--effectiveness analysis of the 
entire evaluation package. Interim reports on 
progress will be issued throughout this pharse and 
feedback solicited from interested colleagues. 
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